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The  Problem  Being  Addressed 


NAVAIR  was  looking  for  a  cost-effective  way  to 
focus  its  process  improvement  efforts: 

•  Many  projects  are  small  and  cannot  afford  the  cost 
and  time  investment  of  an  extensive  SCAMPI 

•  For  projects  that  achieved  a  formal  rating,  we’d  like 
to  do  a  quick  look  years  after  the  formal  appraisal 

•  For  teams  large  and  small,  we’d  like  to  do  more 
than  just  identify  if  they  have  a  process  and  follow  it 

•  We’d  like  to  help  identify  strengths  and  weaknesses 
of  the  processes  being  used  throughout  NAVAIR 

•  We’d  like  to  learn  from  the  stronger  teams,  and 
share  good  practices  with  teams  that  need  help 
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Solution  to  the  Problem 


Our  internally  developed  Model  Based  Appraisal 
(MBA)  method  solved  our  problem  by: 

•  Providing  a  low-cost,  quick,  effective,  accurate,  and 
consistent  appraisal  method  with  minimal 
interruptions  to  the  team 

•  Using  a  reference  model  already  accepted  by 
NAVAIR:  CMMI 

•  Helping  to  establish  a  baseline  against  which  to 
measure  future  process  improvement 

•  Providing  actionable  improvement  opportunities  to 
the  appraised  teams 

•  Capturing  process  vulnerabilities  and  improvement 
opportunities  across  project  teams 
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The  MBA  Process:  Planning  &  Preparation 


Each  Appraisal  starts  with  a  Planning  Phase: 

•  MBA  Lead  works  with  sponsor  to  develop  Appraisal 
Plan  focusing  on  the  model  and  org  scope. 

•  Because  a  consistent  model  scope  was  desired  across 
NAVAIR,  planning  could  be  simplified  somewhat. 

•  Data  Collection  plans  adjusted  to  match  expected 
availability  of  artifacts  from  the  selected  group 

Next,  Appraisal  Preparation  &  Evidence  Collection: 

•  The  group  being  appraised  provides  agreed-upon 
artifacts  to  the  MBA  team 

•  MBA  team  reviews  artifacts  and  develops  interview 
questions 

•  MBA  team  conducts  interviews  to  fill  in  the  gaps  left 
after  reviewing  artifacts 
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The  MBA  Process:  Scoring 


Documentary  and  interview  evidence  for  each  in-scope 
process  area  (PA)  at  the  practice  level  is  entered  into 
the  observation  workbook. 

Scores  are  assigned  to  each  goal  which  is  comprised 
of  related  practices.  These  scores  are  reflected  in  a 
grid  chart  commonly  known  as  a  quilt  chart. 

Goals  are  scored  to  help  identify  which  teams  need 
help  in  which  PAs  using  the  CMMI  as  the  reference 
model.  Two  scores  are  given: 

•  “C”  means  they  have  coverage  and  are  doing 
pretty  well  in  this  process  area 

•  “L”  means  they  lack  coverage  and  there  is  room 
for  improvement  with  practices  related  to  this  PA. 
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The  MBA  Process:  goal  characterization 


ID 

Goals/Practices 

Document  Title  or 
Description  (linked) 

Mapping  Notes 

Direct,  Indirect, 

or 

Process  Asset 

Goal 

Score 

SGI 

Requirements  are  managed  and 
inconsistencies  with  project  plans  and 
work  products  are  identified. 

C 

SP1.1 

Develop  an  understanding  with  the 
requirements  providers  on  the  meaning  of 
the  requirements. 

Software  Requirements 
Specification  (Software 
Soecification  Aloha  Tanqorfl^^ 

customer  sign-off  via 

1 1  nrnil 

Direct 

SP1.2 

Obtain  commitment  to  the  requirements 
from  the  project  participants. 

7  J 

SP1.3 

Manage  changes  to  the  requirements  as 
they  evolve  during  the  project. 

Demo  of  RequisV^ 

Alpha  Tango  requfFTJliiuiu  - 
actively  tracked  and  updated 

Direct 

SP1.4 

Maintain  bi-directional  traceability  among 
the  requirements  and  the  project  plans 
and  work  products. 

Demo  of  Requisite  Pro  showed 
Alpha  Tango  requirements 
traceability 

ability  to  trace  from 
direct  and  derived 
requirements  to  design 
components  and  then  to 
code  components 

Direct 

SP1.5 

Identify  inconsistencies  between  the 
project  plans  and  work  products  and  the 
requirements. 

Email  notes  from  engineers  to 
PM  identifying  omission  of 
project  tasking  needed  to  fulfill 
requirements 

Design  components 
failed  to  address  two 
requirements  and  emails 
requested  additional  time 
to  revise  the  design  to 
accommodate 

Indirect 
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The  MBA  Process:  goal  characterization 


ID 

Goals/Practices 

Document  Title  or 
Description  (linked) 

Mapping  Notes 

Direct,  Indirect, 

or 

Process  Asset 

Goal 

Score 

_  _  . 

SGI  Requirements  are  managed  and 

inconsistencies  with  project  plans  and 
work  products  are  identified. 

r  c 

SP1.1 

Develop  an  understanding  with  the 
r — [iiiri — amyiders  on  the  meaning  of 

Software  Requirements 
Specification  (Software 
Specification  Aloha  Tanao.doc) 

customer  sign-off  vig,,*. 

Direct 

Jb  the  requirements 
|  Vficipants. 

S>^ 

„ I*  Jlanges  to  the  requirements  as 
they  evolve  during  the  project. 

*l5erno  of  Requisite  Pro  showed 
Alpha  Tango  requirements 
actively  tracked  and  updated 

Direct 

Goal  Scoring  is  subjectively  determined  by  trained  process  personnel  who 
consider  the  quantity  and  quality  of  evidence  available  for  each  SG.  Based 
upon  the  evidence,  the  team  gauges  the  extent  to  which  the  appraised  team 
is  implementing  processes  toward  the  achievement  of  each  SG. 

_  Sufficient  evidence  exists  to  indicate  that  the  intent  of  the  goal  is  being  addressed 
_  Sufficient  evidence  does  not  exist  to  indicate  that  the  intent  of  the  goal  is  being  addressed 

NAV^AI  R 
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MBA  Outputs:  goal  characterization 


CMMI  Maturity  Level  2  Goals 


h~~~~'>^Goals 

SGI 

SG2 

Process  Area 

REQM 

C 

PP 

c 

L 

PMC 

c 

L 

SAM 

L 

L 

MA 

L 

C 

PPQA 

C 

L 

CM 

C 

C 

SG3 


GG2 


Sufficient  evidence  exists  to  indicate  that  the  intent  of  the  goal  is  being  addressed 
Sufficient  evidence  does  not  exist  to  indicate  that  the  intent  of  the  goal  is  being  addressed 
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MBA  Outputs:  Suggested  Actions  Report 


Suggestion 

Expected  Benefit/Rationale 

CMMI 

Reference 

Establish  a  standard  for 
which  sections  of  the  SCR 
and  Work  Package  forms 
(for  instance,  the 
modules/functionality 
affected  check  box)  must 
be  completed,  when  and 
by  whom. 

Ensure  that  all  pertinent  data  is 
captured  for  future  reference.  After 
some  amount  of  historical  data  has 
been  captured,  a  simple  but  effective 
analysis  can  be  generated  to  quantify 
the  impact  of  various  estimating 
attributes  that  can  be  used  for  future 
estimating  efforts. 

PPSP  1.2 

Establish  and  reinforce 
policy/training  for  proper 
handling  of  products. 

Training  personnel  in  the  proper 
handling  of  products  in  the  lab  and 
products  in  transition  to  the  customer 
will  reduce  the  risk  of  product  failures. 

SAM  SP  2.4 

Document  training  plans 

Process  OJT  suffers  degradation  as  it 
passes  from  person  to  person  if  there  is 
no  outline  of  critical  information  that 
must  be  maintained. 

GP  2.5 
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MBA  Outputs:  Suggested  Actions  Report 


•  The  suggested  action  report  provides  actionable 
recommendations  to  the  appraised  group. 

•  Many  suggested  actions  can  be  re-used  across 
multiple  teams,  as  a  result,  overtime,  creating  the 
report  doesn’t  take  as  much  time  as  you  might  think. 

•  When  teams  act  on  the  suggested  actions,  resulting 
solutions  can  be  shared  with  other  teams  with  similar 
needs 
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How  MBA  Results  Can  Be  Used 


Project: 

•  Project  team  members  and  other  stakeholders  or  their 
management  receive  a  brief  of  the  Final  Findings  and 
the  Suggested  Actions  Report. 

•  Final  Findings  identify  what  was  found.  The  Suggested 
Actions  report  explains  recommendations  for  change 
and  what  benefits  can  be  expected  from  making  each 
change. 

•  The  Suggested  Actions  report  addresses  every  “L”  on 
the  quilt  chart  so  the  severity  of  the  gaps  is  known. 

•  The  report  may  also  address  a  “C”  if  there  is  a  good 
opportunity  for  further  improvement  in  that  area. 
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How  MBA  Results  Can  Be  Used 


Product  Team: 

•  The  recommendations  in  the  Suggested  Actions  Report 
may  be  applicable  to  all  projects  within  the  product  team 
that  perform  similar  work  and  use  the  same  processes 

•  The  product  team  will  be  able  to  select  the  process 
improvements  that  make  the  most  sense  to  them 

•  This  determination  should  be  made  based  upon  a 
combination  of  business  priorities,  benefits  they’ll 
receive  from  the  change,  and  ability  to  implement  the 
change  (i.e.,  resource  availability,  skills,  cost,  etc.). 

•  Product  teams  need  to  encourage  their  projects  to  act 
upon  the  findings,  and  actually  implement  the  suggested 
changes  that  are  deemed  appropriate 
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How  MBA  Results  Can  Be  Used 


SSEPG  for  Product  Teams: 


•  The  “L”s  in  the  aggregated  quilt  chart  will  allow  the 
SSEPG  to  identify,  by  PA,  where  an  organization’s 
teams  need  the  most  help. 

•  The  SSEPG  can  make  a  first  pass  and  select  the  PAs 
with  the  highest  number  of  “L”s. 

•  In  a  second  pass,  it  can  investigate  these  select  PAs  in 
greater  depth  in  order  to  determine  root  causes  of  the 
problems.  (A  common  root  cause  can  be  addressed  by 
one  solution  that  can  be  shared  across  product  teams.) 

•  The  SSEPG  can  then  identify  high  ROI  opportunities 
across  teams  where  process  performance  is  low  and 
there  appears  to  be  a  common  root  cause. 
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How  MBA  Results  Can  Be  Used 

Sample  SSEPG  Analysis 
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How  MBA  Results  Can  Be  Used 


Organizational  Leadership: 

•  The  organizational  leaders  will  see  the  aggregated 
quilt  chart  and  sanitized  analyses  from  individual 
product  team  SSEPGs 

•  The  leaders  get  a  factual  Model  Based  Appraisal 
based  upon  a  sampling  of  projects  within  the  org. 

This  baseline  serves  as  a  yardstick  against  which  to 
evaluate  PI  progress  (and  ROI)  in  the  future. 

•  The  present  value  of  the  baseline  is  a  foundation  for 
discussions  pertaining  to  desired  capability  levels 
(e.g.,  target  capability  profiles). 

•  The  baseline  will  help  the  organization  and  individual 
SSEPGs  communicate  better  concerning  PI  priorities. 
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Appraisal  Issues  to  Consider 


•  Recommendations  -  Delivery  of  a  Suggested  Actions  Report 
is  fairly  unique  to  our  MBAs,  but  is  extremely  useful 

•  Planning  -  Planning  must  be  thorough  to  ensure  appraisal 
stays  focused  on  the  team’s  most  meaningful  processes 

•  Level  of  rigor  -  Appraiser  should  look  at  the  entire  picture, 
review  the  evidence,  and  talk  to  the  process  participants 

•  Quantity/depth  of  evidence  reviewed  -  A  quick  look  at  a  few 
pieces  of  evidence  may  not  be  sufficient  to  provide  a  good  set 
of  actionable  recommendations  to  the  appraised  organization 

•  Quantity/quality  of  Final  Findings  -  Details  of  any  appraisal 
that  will  go  beyond  the  immediate  team  must  be  balanced  by 
appraisal  principles  i.e.  anonymity  and  non-attribution 
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Interesting  Lessons  Learned 


•  The  shorter  the  prep  time  for  interviewed  employees 
the  more  realistic  of  a  snapshot  will  be  obtained 

•  Remote  MBAs  are  a  viable  option  which  can  enable 
the  professionals  to  do  more  appraisals  efficiently 

•  Remote  MBAs  enable  team  members  who  are  on 
travel  (or  even  on  vacation)  to  be  interviewed  with  the 
same  ease  as  if  they  were  present 

•  If  interviews  cannot  be  done  at  the  scheduled  date, 
there  is  less  impact  to  the  process  with  remote  MBAs 
since  completion  schedule  can  be  more  flexible 

•  The  above  lessons  are  what  enable  the  MBA  to  be 
performed  at  low  cost  and  low  impact  to  the  projects 
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Conclusions 


•  MBAs  enable  the  product  team  or  organization  SSEPGs 
to  identify,  prioritize  and  correct  process  deficiencies  at 
various  levels 

•  MBAs  provide  actionable  process  improvements 
relevant  to  the  appraised  organizations 

•  MBAs  provide  a  snapshot  of  process  maturity  across 
the  organization  so  improvements  can  be  measured 

•  Team  should  fully  and  enthusiastically  support  this  MBA 
approach  because  of  the  potential  return  on  investment 

•  Organizations  should  establish  policy  to  follow-up  and 
act  upon  the  findings  and  recommendations  generated 
from  the  MBAs 
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